How can we speak math?

It is likely that most people can communicate mathematics to a computer more effectively (rapidly and accurately) by speaking than they can by using a stylus on a computer tablet. This may seem surprising, but is our speculation based on trying various alternative input methods. An even better setup may be to speak and simultaneously use pointing or handwriting. Unfortunately, building a properly functioning prototype using this concept is difficult. Yet a successful implementation of such a “multimodal” combination should allow the computer to reinforce correct recognition while identifying and perhaps repairing “unimodal” errors. In some cases speaking may be more convenient than typing, even for rapid typists: many mathematical symbols are missing from the keyboard but can be easily spoken and recognized. Even without venturing into Greek, or alternative fonts, just handwriting or even typing a number, say “fifty million” may be slower and more error-prone than speaking. Pursuing the goal of effectively speaking and recognizing small pieces of mathematics, oed to a study of how hard it would be to speak arbitrarily long sections of mathematics, including nested complex expressions. We first describe programs for the inverse problem: computer generation of mathematical speech. This requires that we address some speaking conventions to overcome the unfortunately ambiguous and inconsistent common usages of mathematics. Then we consider tools and guidelines to make it more plausible for humans to speak full mathematical formulas unambiguously so they can be recognized by a computer using a speech recognizer program. We describe our prototype programs which do somewhat less than we propose, but are effective in that speech can either be used alone, or used to fill in boxes (superscripts, etc.) or larger pieces. Speech can also be used for choosing alternatives from plausible symbols resulting from uncertain recognition from handwriting (or speech). We believe the principal barriers to engineering a more complete program can be overcome, though a driving application may be essential for refining prototypes into useful programs. This paper is not intended to be the last word on the subject, but simply exposes problems and approaches relevant to the task. Demonstrations of partial implementations are available as Window (XP) programs.